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(57) Abstract: A method and a system to simulate 
ultrasound images and hence for simulating the evaluation 
of a subject with such images as a diagnostic tool. For the 
purpose of simulating ultrasound images, these images 
may be considered to be planar slices of a given geometry. 
These slices are generated from the surface geometry 
of the portion of the subject which would receive the 
ultrasound waves if the ultrasound diagnostic procedure 
was actually be performed. Simulating such ultrasound 
images may therefore be performed by determining such 
images from surface models, and then rendering those 
ultrasound images at the desired position. The rendering 
of such simulated ultrasound images also optionally and 
preferably includes the simulation of the "grainy" quality 
of such images. 
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FIELD OF THE INVENTION 
5 The present invention is of a method for simulating ultrasound images, and in particular, 

for such a method for simulating ultrasound images which are produced during a medical 
diagnostic procedure. 

BACKGROUND OF THE INVENTION 
10 ERCP (Endoscopic Retrograde Cholangio Pancreatography) is a minimally invasive 

procedure which allows the doctor to perform necessary treatments such as enlarging a bile duct 
opening, removing gallstones lodged in the bile duct, inserting a stent (drain) in the duct or 
taking a biopsy specimen. 

A flexible fiber optic tube, called a duodenoscope, is passed through the mouth, 
15 esophagus and stomach into the duodenum, which is the first part of the small intestine. The 
duodenoscope is a thin, flexible tube with a tiny video camera and light at one side of the tip. 
The papilla, which is an opening where the bile and pancreatic ducts empty into the duodenum, 
is visually identified. A small plastic tube, or cannula, is passed through the duodenoscope into 
the papilla. X-ray dye is injected through the cannula into the ducts. X-rays are then taken to 
20 study the ducts. Any necessary treatments can be performed at this time. 

This treatment scheme is somewhat similar to gastro-endoseopy, in which an endoscope 
is inserted through the rectum in order to examine the large colon, for example. In both cases, 
students are taught to perform such procedures according to the traditional model for medical 
education, in which students observe and assist more experienced physicians. Unfortunately, 
25 such observation alone cannot provide the necessary training for such complicated medical 

procedures. Students may also perform procedures on animals and human cadavers, neither of 
which replicates the visual and tactile sensations of a live human patient. Thus, traditional 
medical training is not adequate for modern technologically complex medical procedures. 

In an attempt to provide more realistic medical training for such procedures, simulation 
30 devices have been developed which attempt to replicate the tactile sensations and/or visual 
feedback for these procedures, in order to provide improved medical training without 
endangering human patients. An example of such a simulation device is disclosed in U.S. Patent 
No. 5,403,191, in which the disclosed device is a box containing simulated human organs. 
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Various surgical laparoscopic procedures can be performed on the simulated organs. Visual 
feedback is provided by a system of mirrors. However, the system of both visual and tactile 
feedback is primitive in this device, and does not provide a true representation of the visual and 
tactile sensations which would accompany such surgical procedures in a human patient. 
Furthermore, the box itself is not a realistic representation of the three-dimensional structure of a 
human patient. Thus, the disclosed device is lacking in many important aspects and fails to meet 
the needs of a medical simulation device. 

Attempts to provide a more realistic experience from a medical simulation device are 
disclosed in PCT Patent Application Nos. WO 96/16389 and WO 95/02233. Both of these 
applications disclose a device for providing a simulation of the surgical procedure of 
laparoscopy. Both devices include a mannequin in the shape of a human torso, with various 
points at which simulated surgical instruments are placed. However, the devices are limited in 
that the positions of the simulated surgical instruments are predetermined, which is not a realistic 
scenario. Furthermore, the visual feedback is based upon a stream of video images taken from 
actual surgical procedures. However, such simple rendering of video images would result in 
inaccurate or unrealistic images as portions of the video data would need to be removed for 
greater processing speed. Alternatively, the video processing would consume such massive 
amounts of computational time and resources that the entire system would fail to respond in a 
realistic time period to the actions of the student. At the very m i nim u m , a dedicated graphics . 
workstation would be required, rather than a personal computer (PC). Thus, neither reference 
teaches or discloses adequate visual processing for real time visual feedback of the simulated 
medical procedure. 

Similarly, U.S. Patent No. 4,907,973 discloses a device for simulating the medical 
procedure of flexible gastro-endoscopy. The disclosed device also suffers from the deficiencies 
of the above-referenced prior art devices, in that the visual feedback system is based upon 
rendering of video data taken from actual duodenoscopic procedures. As noted previously, 
displaying such data would either require massive computational resources, or else would simply 
require too much time for a realistic visual feedback response. Thus, the disclosed device also 
suffers from the deficiencies of the prior art. 

A more useful and efficient medical simulation device for minimally invasive therapeutic 
procedures such as endoscopy is disclosed in PCT Application No. WO 99/38141, by the present 
inventors and incorporated by reference as if fully set forth herein. The disclosed medical 
simulation device provides real time, accurate and realistic visual feedback of general 
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endoscopic procedures, as well as realistic tactile feedback, so that the visual and tactile systems 
are accurately linked for the simulation as for an actual medical procedure. 

Duodenoscopy of the bilio-pancreatic system would also benefit from such realistic 
simulation, involving both visual and tactile feedback which are provided in an accurate manner. 
Bilio-pancreatic duodenoscopic procedures feature many of the same principles as gastro- 
endoscopy, since for both types of endoscopic procedures, an instrument is inserted into a body 
orifice, and must then be guided through a tubular organ without direct visual feedback. In 
addition, the physician performing the procedure must be able to correctly interpret both the 
indirect visual feedback provided through a video monitor, as well as the tactile feedback 
through the instrument itself. Therefore, both types of duodenoscopy require the physician to 
receive "hands-on" manual training for the correct performance of the procedure. 

As part of the simulation of this procedure, the students should also be able to practice 
using the diagnostic tool of ultrasound. Ultrasonography is an optional adjunct to a large 
number of medical procedures, such as gastro-endoscopy and duodenoscopy. In addition, 
ultrasonography is itself an important medical procedure, which is used for example in order to 
safely visualize the fetus during pregnancy. Unfortunately, the background art does not feature 
suitable methods for accurately and efficiently simulating ultrasound images. 

ST TMMARY nv TTTF INVENTION 

The background art does not teach or suggest a method and a system to efficiently and 
realistically simulate ultrasound images. Nor does the background art teach or suggest such a 
system and method for providing such images as part of the simulation of an overall medical 
procedure. 

The present invention overcomes the deficiencies of the background art by providing a 
method and a system to simulate ultrasound images and hence for simulating the evaluation of a 
subject with such images as a diagnostic tool. For the purpose of simulating ultrasound images, 
these images may be considered to be planar slices of a given geometry. These slices are 
generated from the surface geometry of the portion of the subject which would receive the 
ultrasound waves if the ultrasound diagnostic procedure was actually being performed. 
Simulating such ultrasound images may therefore be performed by determining such images 
from surface models, and then rendering those ultrasound images at positions given by the tip of 
the scope. The rendering of such simulated ultrasound images also optionally and preferably 
includes the simulation of the "grainy" quality of such images. 
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According to the present invention, there is provided a method for simulating an 
ultrasound image of at least one anatomical feature of a subject, comprising: determining a 
surface geometry of the at least one anatomical feature; generating at least one planar slice of the 
surface geometry according to a location of a simulated source of ultrasound waves; and 
rendering the ultrasound image according to the at least one planar slice. 

Hereinafter, the term "simulated medical procedure" refers to the simulation of the 
medical procedure as performed through the system and method of the present invention. 
Hereinafter, the term "actual medical procedure" refers to the performance of the medical 
procedure on an actual, living human patient with an actual medical instrument, such that the 
medical procedure is "real" rather than "simulated". Hereinafter, the term "corresponding actual 
organ" refers to the "real" organ of a human being or other mammal which is being simulated by 
the simulated organ of the present invention. 

Hereinafter, the term "subject" refers to the human or lower mammal upon which the 
method and system of the present invention are performed or operated. Hereinafter, the term 
"student" refers to any human using the system of the present invention, being trained according 
to the present invention or being taught according to the present invention including, but not 
limited to, students attending medical school or a university, a medical doctor, a trained 
pancreatic specialist or other trained medical specialist. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, aspects and advantages will be better understood from 
the following detailed description of a preferred embodiment of the invention with reference to 

the drawings, wherein: 

FIG. 1 is a flowchart of an exemplary method of the present invention; 

FIG. 2 is a flow diagram for determining the output texture for ultrasound images 
according to the present invention; 

FIG. 3 shows exemplary ultrasound texture which is simulated according to the present 

invention; 

FIG. 4 shows labeled exemplary organ slices, visualized through ultrasound, which are 
simulated according to the present invention; and 

FIG. 5 shows an exemplary final ultrasound image, simulated according to the method of 

the present invention. 
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PET ATT .ED DESCRIPTTON OF PREFERRED EMBODIMENTS 

The present invention is related to a method and a system for simulating ultrasound 
images and hence for simulating the evaluation of a subject with such images as a diagnostic tool. 
Preferably, but not necessarily, the ultrasound images are simulated as though they had been 
5 collected during the evaluation of a subject with gastro-endoscopy and/or duodenoscopy, 
although of course ultrasound images could optionally be simulated as though they had been 
collected during substantially any medical procedure. 

For the purpose of simulating ultrasound images, these images may be considered to be 
planar slices of a given geometry. These slices are generated from the surface geometry of the 
10 portion of the subject which would receive the ultrasound waves if the ultrasound diagnostic 
procedure was actually being performed. This surface geometry may optionally be modeled as 
polygonal models or other surface representation, optionally and more preferably by using 
common 3D graphic accelerator functionality. Thus, a real-time performance to this simulation 
may optionally be obtained. 
15 Simulating such ultrasound images may therefore be performed by determining such 

images from surface models, and then rendering those ultrasound images at positions given by 
the tip of the scope. Such rendering is similar to that currently performed for modeling regular 
endoscopy procedures, and is also known as a "movement model". One example of such a 
movement model is given in PCT Application No. WO 99/38141, filed on January 15, 1999, 
20 which is hereby incorporated by reference as if folly set forth herein. This reference also 
describes an exemplary system for simulating a medical procedure, which could optionally be 
used with the present invention, for example. Of course, other systems and/or devices could also 
optionally be used with the method of the present invention. 

The present invention differs from such methods of modeling visual data for gastro- 
25 endoscopic procedures in that additional areas of the subject, including additional organs and/or 
organ portions, must also be rendered, since ultrasound waves penetrate beyond the wall of the 
organ which actually contains the ultrasound probe. 

The rendering of such simulated ultrasound images also optionally and preferably 
includes the simulation of the "grainy" quality of such images. This quality is inherent to the 
30 method of ultrasound itself, as it involves sending sound waves to the desired area of the subject 
and then listening to the echoes, in order to measure the time required for the echo to return. Of 
course, at least a portion of the actual signal would be attenuated, since not all of the sound 
waves are reflected by the tissue. The intensity of the echo is also measured to provide the 
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"echoic property" of the object, in which white spots in an image are hyper-echoic, and black 
spots are hypo-echoic. The graininess of the image occurs since the sound waves reflect in all 
directions, so minor echoes that are reflected locations other than the organ being examined also 
return to the probe, causing static interference with the signal being measured. Such static 
5 interference then creates the grainy image. 

The principles and operation of a method and a system according to the present invention 
for the simulation of the medical diagnostic procedure of ultrasound, may be better understood 
with reference to the drawings and the accompanying description, it being understood that these 
drawings are given for illustrative purposes only and are not meant to be limiting. 
10 As shown with regard to Figure 1, an exemplary method of the present invention for 

rendering ultrasound images is described. Although the explanation is given with regard to a 
simulation of a linear array probe, in which the plane of the scan is parallel to the shaft of the 
endoscope probe, the system could easily be used to simulate other types of US (ultrasound) 
scans, for example a radial scan in which the plane of the scan is perpendicular to the shaft of the 
15 endoscope. One of ordinary skill in the art could easily alter the simulation process in order to 
accommodate these differences between scanning devices. 

First, a technique called "stenciling" is used to render a slice of an object from a specific 
plane into a texture (which is basically a bitmap or a digital picture). Stenciling is known in the 
art of real time graphics, and is also used to simulate shadows and create other real-time 
20 computer graphic effects, in conjunction with a stencil buffer. This technique is performed by 
positioning the viewpoint of a virtual "camera" above the desired plane, in the direction of the 
plane normal. Positioning of a virtual camera or viewpoint for generating three-dimensional 
computer graphics is achieved by giving the coordinates of the viewpoint in relation to the 
coordinates of the object being viewed, by using a standard coordinate system (such as that of 
25 the X ,Y & Z Axes for example). 

The near Z clip plane is then set to fit the cutting plane. The near Z clip plane is the 
distance from the camera at which objects begin to be seen, such that objects closer than that 
distance are clipped or not shown. The requirement for the near Z clip plane in the background 
art is due to mathematical inaccuracies in computers, as objects cannot to come infinitely close 
30 to the camera. However, for the purposes of the present invention, this parameter is preferably 
set at a specific location in order to cause the object to be cut at a specific plane, in order to be 
able to render the ultrasound image at that plane. This parameter could optionally be set in any 
type of three-dimensional (3D) rendering software to cause the object to be so clipped. The 
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general function of a near Z clipping plane is to remove all the geometry that is closer to the 
camera than to that plane. 

Next, both sides of the surface of the whole object are preferably rendered. All the 
surfaces of the object are rendered, even those which are not seen such as those which are 
5 removed from view, for example in the event that a near Z clipping plane has been applied. 
When the object is rendered, the stencil buffer associated with that software is preferably used to 
count the amount of pixels written to the buffer. If an even amount of pixels are written, the 
slice is not visible, and if an odd number of pixels are written, the slice is visible, as shown with 
regard to Figure 2. This effect occurs because both sides are rendered, but with different 
10 parameters. For those portions of the object which should not be displayed (such as the back 
part of an object, which could not be seen from the perspective of the user), each time a pixel is 
written to the image buffer, the stencil value for this pixel is incremented. For those portions 
which should be displayed, the value in the stencil buffer is decreased by one. The stencil buffer 
is therefore a bitmap of 0 and 1 values, in which values Of "1" indicate where this object 
1 5 intersects the ultrasound image plane. 

A texture is then preferably selected which resembles the texture of the ultrasound image 
for the desired object, which could optionally be extracted from real ultrasound images of the 
organ or other tissue, such as images of the liver for example. Next, this texture is rendered by 
using the previously defined information from each slice as the alpha channel. The alpha 
20 channel is another color channel which is added frequently in 3D graphics and computer 
graphics generally, in addition to the common Red/Green/Blue channels (or color components). 
The alpha channel is related to the amount of transparency of the color. The data of the fourth 
channel is then preferably applied to all graphic components containing color. With regard to 
Figures 2 and 3, the alpha channel is used to determine which areas of the simulated images 

25 below are transparent. 

With regard to the texture, in order to add visual realism to those portions of image in the 
previously described stencil buffer, texture is preferably added to those portions of the image 
having values of "1 " in that buffer. For the purposes of this description, the texture is preferably 
added in grayscale, although of course colors could optionally be added. 

30 These steps are preferably performed for each object separately until all visible object 

slices are rendered, resulting in a simulated image as shown below with regard to Figure 4. The 
image is then preferably "bent" and blurred to simulate the poor resolution of true ultrasound 
images. The process of "bending" the image is more preferably performed mathematically, while 



WO 02/094080 



PCT/BL02/00393 



8 

blurring may optionally and more preferably be performed through image processing which also 
involves a mathematical function, and which is also very well known in the art. 

All the work done previously is rendered to a surface that is not visible to the user, such 
that there is now a rectangular picture that represents slices of all the objects that intersected the 
5 US plane. If it is taken into consideration that the US probe is not straight, but is in fact an array 
of sensors in the shape of an an arc, the resulting image should be in the shape of a sector 
circumscribed by an arc parallel to the base arc of the sector. To obtain this image the 
rectangular picture is rendered onto a template in the shape of a sector as described above, in 
which the template is a polygonal model that describes the ultimate overall shape of the image. 
10 The result of these processes is shown in the exemplary image of Figure 5. 

After these processes have been performed, the relevant anatomy has been modeled onto 
surface models of the anatomical feature, such as the organ. Each object has been rendered with 
the associated ultrasound properties, such as how the tissue responds to ultrasound waves in 
terms of echoing, and with a sample of the correct filling texture for simulation of the ultrasound 
15 image. For example, since moving the endoscope or other tool which holds the probe also alters 
the probe's position, the position of the cutting plane may be altered with each movement. In 
turn, the slices generated by the cutting plane are changed, such that a new ultrasound image is 
formed. The final simulated ultrasound image is then displayed to the user (see Figure 5 for an 
example). 

20 As previously described, one of ordinary skill in the art could easily adjust the method of 

the present invention in order to accommodate different types of ultrasound scanning 
technologies. For example, for a radial scanning the device, the image would only have "blur" 
added, but would not be "bent", because the plane of scanning is perpendicular to the device. 
Also, the image would be round rather than square. 

25 It will be appreciated that the above descriptions are intended only to serve as examples, 

and that many other embodiments are possible within the spirit and the scope of the present 
invention. 
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WHAT IS CLAIMED IS: 

1 . A method for simulating an ultrasound image of at least one anatomical feature of 

a subject, comprising: 

detennining a surface geometry of the at least one anatomical feature; 

generating at least one planar slice of said surface geometry according to a location of a 
simulated source of ultrasound waves; and 

rendering the ultrasound image according to said at least one planar slice. 

2. The method of claim 1, wherein said generating said at least one planar slice is 
performed by using a stencil buffer. 

3 . The method of claim 2, wherein said generating with said stencil buffer is 
performed by: 

positioning a viewpoint for collecting the ultrasound image above said at least one planar 
slice; and 

setting a near Z clip plane to fit a cutting plane for cutting said at least one planar slice. 

4. The method of claim 3, wherein said generating with said stencil buffer is further 
performed by: 

setting a value for pixels for said planar slice to a positive value, and a value of pixels for 
the remainder of the anatomical feature to a zero value; and 
rendering said pixels. 

5. The method of claim 3, wherein said generating with said stencil buffer is further 
performed by: 

determining at least one intersection of the anatomical feature with said planar slice; and 
setting a value for said pixels in said planar slice to a positive value for said stencil 

buffer. 

6. The method of any of claims 1 -5, wherein the ultrasound image is a grayscale 
image and wherein said rendering further comprises rendering said grayscale image with a 
selected texture, said selected texture being collected from an actual ultrasound medical 
procedure. 
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7. The method of any of claims 1-5, wherein the ultrasound image is a black and 
white image, and wherein said rendering further comprises rendering said black and white image 
with a selected texture, said selected texture being collected from an actual ultrasound medical 
procedure. 

8. The method of either of claims 6 or 7, wherein said rendering with said selected 
texture is only performed for pixels in said stencil buffer having a positive value. 

9. The method of claim 8, wherein said pixels having said positive value are white 

pixels. 

10. The method of claims 6 or 7, wherein a simulated ultrasound image is created by 
rendering said texture on said at least one planar slice; 

1 1 . The method of claim 10, wherein rendering said texture further comprises at least 
one of bending and blurring said simulated image. 

12. The method of claim 11, wherein said bending further comprises: 
rendering said simulated image onto a template, said template comprising a polygonal 

model constructed according to a shape of an image from an actual ultrasound medical 
procedure . 
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